{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# PaddleCamp - 第二期\n",
    "\n",
    "## 第三周 - 第二次作业\n",
    "\n",
    "* 代码题\n",
    "1. 该部分不计分数；\n",
    "2. 请同学仔细阅读代码及说明，按提示完成迁移学习的练习；\n",
    "3. 同学可尝试改动代码，观察训练效果。\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Trans__learning/data_sets/flower_photos/roses/4860145119_b1c3cbaa4e_n.jpg\r",
      "Trans__learning/data_sets/flower_photos/roses/5223191368_01aedb6547_n.jpg\r",
      "Trans__learning/data_sets/flower_photos/roses/15172358234_28706749a5.jpg\r",
      "Trans__learning/data_sets/flower_photos/roses/5960270643_1b8a94822e_m.jpg\r",
      "Trans__learning/data_sets/flower_photos/roses/3179751458_9646d839f6_n.jpg\r",
      "Trans__learning/data_sets/flower_photos/roses/3026375835_a20ecdd140_m.jpg\r",
      "Trans__learning/data_sets/flower_photos/roses/16152175716_55d6968e08_n.jpg\r",
      "Trans__learning/data_sets/flower_photos/roses/3115889021_053f3b8e5a.jpg\r",
      "Trans__learning/data_sets/flower_photos/roses/5249566718_6109630c83_m.jpg\r",
      "Trans__learning/data_sets/flower_photos/roses/6676529655_9672b6f955_m.jpg\r",
      "Trans__learning/data_sets/flower_photos/roses/527513005_41497ca4dc.jpg\r",
      "Trans__learning/data_sets/flower_photos/roses/14221192676_eb8c89a7d6_n.jpg\r",
      "Trans__learning/data_sets/flower_photos/roses/17990320484_93bba345d2_m.jpg\r",
      "Trans__learning/data_sets/flower_photos/roses/15951588433_c0713cbfc6_m.jpg\r",
      "Trans__learning/data_sets/flower_photos/roses/16051111039_0f0626a241_n.jpg\r",
      "Trans__learning/data_sets/flower_photos/roses/6231418894_7946a7712b_n.jpg\r",
      "Trans__learning/data_sets/flower_photos/roses/3171577977_8608282f04_m.jpg\r",
      "Trans__learning/data_sets/flower_photos/roses/6255593451_b8a3aa8f7a_m.jpg\r",
      "Trans__learning/data_sets/flower_photos/roses/494803274_f84f21d53a.jpg\r",
      "Trans__learning/data_sets/flower_photos/roses/15222804561_0fde5eb4ae_n.jpg\r",
      "Trans__learning/data_sets/flower_photos/roses/6280787884_141cd7b382_n.jpg\r",
      "Trans__learning/data_sets/flower_photos/roses/3072908271_08764c732a_m.jpg\r",
      "Trans__learning/data_sets/flower_photos/roses/1831404161_d2df86fd70.jpg\r",
      "Trans__learning/data_sets/flower_photos/roses/3292434691_392071d702_n.jpg\r",
      "Trans__learning/data_sets/flower_photos/roses/5628552852_60bbe8d9b0_n.jpg\r",
      "Trans__learning/data_sets/flower_photos/roses/4231745228_ece86330d9.jpg\r",
      "Trans__learning/data_sets/flower_photos/roses/3921794817_276eb4386b.jpg\r",
      "Trans__learning/reader.py\r",
      "Trans__learning/.ipynb_checkpoints/\r",
      "Trans__learning/.ipynb_checkpoints/trans__learning__finetune-checkpoint.ipynb\r",
      "Trans__learning/models/\r",
      "Trans__learning/trans__learning__finetune.ipynb\r",
      "Trans__learning/images_jupyter/\r",
      "Trans__learning/images_jupyter/no_yuxunlian.png\r",
      "Trans__learning/images_jupyter/flowers_1.png\r",
      "Trans__learning/images_jupyter/step-1-step-2.png\r",
      "Trans__learning/images_jupyter/train_4.png\r",
      "Trans__learning/images_jupyter/flowers_0.png\r",
      "Trans__learning/images_jupyter/freeze_step1.png\r",
      "Trans__learning/images_jupyter/train_3.png\r",
      "Trans__learning/images_jupyter/train_2.png\r",
      "Trans__learning/images_jupyter/no_freeze_step-1.png\r",
      "Trans__learning/images_jupyter/train_1.png\r",
      "Trans__learning/images_jupyter/trans_learning_1.png\r",
      "Trans__learning/images_jupyter/flowers_2.png\r"
     ]
    }
   ],
   "source": [
    "# 解压必要的文件。第一次的时候请正常运行。之后可以注释掉该 cell 代码\n",
    "!DATA_PATH=data/data7826/ && NEW_NAME=$(find -name *[0-9].ipynb) && NEW_NAME=${NEW_NAME%.*} && NEW_NAME=${NEW_NAME#./} && tar -zxvf ${DATA_PATH}Trans__learning.tar  && cp -rf Trans__learning/. ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# 用迁移学习思想——实现图像分类\n",
    "\n",
    "本次课，我们用迁移学习思想，结合paddle框架，来实现图像的分类。\n",
    "\n",
    "\n",
    "# 相关理论：\n",
    "\n",
    "### 接下来介绍两种迁移学习思想：\n",
    "\n",
    "#### 1. 原有模型作为一个特征提取器:\n",
    "使用一个用ImageNet数据集提前训练（pre-trained)好的CNN，再除去最后一层全连接层(fully-connected layer)，即除去原有的分类器部分。然后再用剩下的神经网络作为特征提取器应用在新的数据集上。我们只需要用新的训练集训练一个嫁接到这个特征提取器上的分类器即可。\n",
    "\n",
    "#### 2.fine-turning原有模型\n",
    "\n",
    "这种方法要求不仅仅除去原有模型的分类器，再用新的数据集训练新的分类器，还要求微调（fine-tune）本身神经网络的参数（weights）。和方式1的区别是：方式1要求固定原有神经网络的参数，而fine-tune允许训练新的数据集时对原有神经网络的参数进行更改。为了防止过拟合，有时可以要求固定某些层的参数，只微调剩下的一些层。\n",
    "\n",
    "两种方式如下图所示：\n",
    "<img src=\"images_jupyter/trans_learning_1.png\" width=\"600\"><br/>"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "- 本次实验我们总共分两个**steps**；  \n",
    "- **step-1:** 在step-1中，我们加载在imagenet数据集上训练好的Resnet50模型，作为我们的预训练模型，并且冻结除fc层之外的参数，只训练fc层。得到step_1_model; \n",
    "\n",
    "当我们拿到新数据集，想要用预训练模型处理的时候，通常大家都会先用step-1的方法看看预训练模型在新数据上的表现怎么样，摸个底。如果表现不错，还想看看能不能进一步提升，就可以试试Fine-tune（即解锁比较少的卷积层继续训练），但是不要期待会有什么质的飞跃。如果由于新数据集与原数据集（例如ImageNet数据集）的差别太大导致表现很糟，那么一方面可以考虑自己从头训练模型，另一方面也可以考虑解锁比较多层的训练，亦或干脆只用预训练模型的参数作为初始值，对模型进行完整训练。 \n",
    "\n",
    "\n",
    "- **step-2:** 在step-2中，我们把step_1_model作为我们的预训练模型，并在此基础上重新训练，得到最终的模型step_2_model。\n",
    "\n",
    "\n",
    "**但是要注意：**   \n",
    "事实上，step-2必须在已经进行过‘冻结特征提取器参数的训练’之后再尝试训练模型，这时分类器的参数已经经过一次训练。如果从随机生成的分类器参数开始直接训练，在做参数更新迭代过程中梯度将很可能过大，而导致模型的崩溃，使模型忘记学到的所有东西。\n",
    "\n",
    "在finetune时，batch_size1设置最好不要太大，以便于加速模型收敛。学习率也适当小一些，\n",
    "\n",
    "接下来我们就进行试验：  \n",
    "\n",
    "# step-1：\n",
    "\n",
    "在step-1中，我们加载在imagenet数据集上训练好的Resnet50模型，作为我们的预训练模型，并且冻结除fc层之外的参数，只训练fc层。得到step_1_model;  "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 1. 导入库\n",
    "实验第一步，我们还是导入我们需要的库，本实验中我们专门定义了一个 reader.py文件，用来对数据集进行读取和预处理，所以我们也需要把reader.py文件import进来。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import shutil\n",
    "import paddle as paddle\n",
    "import paddle.fluid as fluid\n",
    "from paddle.fluid.param_attr import ParamAttr\n",
    "import reader\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "我们利用paddle.batch()获取flowers数据集。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "done\n"
     ]
    }
   ],
   "source": [
    "# 获取flowers数据\n",
    "train_reader = paddle.batch(reader.train(), batch_size=16)\n",
    "test_reader = paddle.batch(reader.val(), batch_size=16)\n",
    "print(\"done\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 2. 定义ResNet网络\n",
    "\n",
    "> - 本次实验我们使用ResNet50这个残差神经网络，所以，接下来我们需要定义一个残差神经网络。  \n",
    "> - PaddlePaddle官方已经提供了ResNet以及其他经典的网络模型，大家可以使用。链接：https://github.com/PaddlePaddle/models/blob/develop/PaddleCV/image_classification/models/resnet.py  \n",
    "> - 网络定义时，每一个层都由指定参数名字。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "done\n"
     ]
    }
   ],
   "source": [
    "# 定义残差神经网络（ResNet）\n",
    "def resnet50(input):\n",
    "    def conv_bn_layer(input, num_filters, filter_size, stride=1, groups=1, act=None, name=None):\n",
    "        conv = fluid.layers.conv2d(input=input,\n",
    "                                   num_filters=num_filters,\n",
    "                                   filter_size=filter_size,\n",
    "                                   stride=stride,\n",
    "                                   padding=(filter_size - 1) // 2,\n",
    "                                   groups=groups,\n",
    "                                   act=None,\n",
    "                                   param_attr=ParamAttr(name=name + \"_weights\"),\n",
    "                                   bias_attr=False,\n",
    "                                   name=name + '.conv2d.output.1')\n",
    "        if name == \"conv1\":\n",
    "            bn_name = \"bn_\" + name\n",
    "        else:\n",
    "            bn_name = \"bn\" + name[3:]\n",
    "        return fluid.layers.batch_norm(input=conv,\n",
    "                                       act=act,\n",
    "                                       name=bn_name + '.output.1',\n",
    "                                       param_attr=ParamAttr(name=bn_name + '_scale'),\n",
    "                                       bias_attr=ParamAttr(bn_name + '_offset'),\n",
    "                                       moving_mean_name=bn_name + '_mean',\n",
    "                                       moving_variance_name=bn_name + '_variance', )\n",
    "\n",
    "    def shortcut(input, ch_out, stride, name):\n",
    "        ch_in = input.shape[1]\n",
    "        if ch_in != ch_out or stride != 1:\n",
    "            return conv_bn_layer(input, ch_out, 1, stride, name=name)\n",
    "        else:\n",
    "            return input\n",
    "\n",
    "    def bottleneck_block(input, num_filters, stride, name):\n",
    "        conv0 = conv_bn_layer(input=input,\n",
    "                              num_filters=num_filters,\n",
    "                              filter_size=1,\n",
    "                              act='relu',\n",
    "                              name=name + \"_branch2a\")\n",
    "        conv1 = conv_bn_layer(input=conv0,\n",
    "                              num_filters=num_filters,\n",
    "                              filter_size=3,\n",
    "                              stride=stride,\n",
    "                              act='relu',\n",
    "                              name=name + \"_branch2b\")\n",
    "        conv2 = conv_bn_layer(input=conv1,\n",
    "                              num_filters=num_filters * 4,\n",
    "                              filter_size=1,\n",
    "                              act=None,\n",
    "                              name=name + \"_branch2c\")\n",
    "\n",
    "        short = shortcut(input, num_filters * 4, stride, name=name + \"_branch1\")\n",
    "\n",
    "        return fluid.layers.elementwise_add(x=short, y=conv2, act='relu', name=name + \".add.output.5\")\n",
    "\n",
    "    depth = [3, 4, 6, 3]\n",
    "    num_filters = [64, 128, 256, 512]\n",
    "\n",
    "    conv = conv_bn_layer(input=input, num_filters=64, filter_size=7, stride=2, act='relu', name=\"conv1\")\n",
    "    conv = fluid.layers.pool2d(input=conv, pool_size=3, pool_stride=2, pool_padding=1, pool_type='max')\n",
    "\n",
    "    for block in range(len(depth)):\n",
    "        for i in range(depth[block]):\n",
    "            conv_name = \"res\" + str(block + 2) + chr(97 + i)\n",
    "            conv = bottleneck_block(input=conv,\n",
    "                                    num_filters=num_filters[block],\n",
    "                                    stride=2 if i == 0 and block != 0 else 1,\n",
    "                                    name=conv_name)\n",
    "\n",
    "    pool = fluid.layers.pool2d(input=conv, pool_size=7, pool_type='avg', global_pooling=True)\n",
    "    return pool\n",
    "print(\"done\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "本次实验，我们使用的图片数据集是flowers。图片是3通道宽高都是224的彩色图，总类别是5种，每一个种类大约有六百多张。  \n",
    "\n",
    "<img src=\"images_jupyter/flowers_0.png\" width=\"600\"><br/>\n",
    "图片示例如下：\n",
    "<img src=\"images_jupyter/flowers_1.png\" width=\"600\"><br/>\n",
    "数据集的标签文件介绍如下：\n",
    "<img src=\"images_jupyter/flowers_2.png\" width=\"600\"><br/>\n",
    "\n",
    "## 3.训练前准备\n",
    "\n",
    "接下来，我们开始做训练前的准备工作：  \n",
    "\n",
    "首先，定义图片数据和标签数据的输入层："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 定义输入层\n",
    "image = fluid.layers.data(name='image', shape=[3, 224, 224], dtype='float32')\n",
    "label = fluid.layers.data(name='label', shape=[1], dtype='int64')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "我们利用stop_gradient，使得pool以上的层停止梯度传递，相当于keras中的freeze。这样我们就可以只训练最后的fc层，但是要注意：我们的数据集是5分类，所以size要设为5."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 获取分类器的上一层\n",
    "pool = resnet50(image)\n",
    "# 停止梯度下降\n",
    "pool.stop_gradient = True\n",
    "# 由这里创建一个基本的主程序\n",
    "base_model_program = fluid.default_main_program().clone()\n",
    "\n",
    "# 这里再重新加载网络的分类器，大小为本项目的分类大小\n",
    "model = fluid.layers.fc(input=pool, size=5, act='softmax')\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "接下来，我们要做的工作有：\n",
    "\n",
    "> - 定义损失函数;  \n",
    "> - 求准确率；  \n",
    "> - 定义优化器；    \n",
    "> - 设定训练场所；  \n",
    "> - 定义执行器，并且完成参数初始化；"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 获取损失函数和准确率函数\n",
    "cost = fluid.layers.cross_entropy(input=model, label=label)\n",
    "avg_cost = fluid.layers.mean(cost)\n",
    "acc = fluid.layers.accuracy(input=model, label=label)\n",
    "\n",
    "# 定义优化方法\n",
    "optimizer = fluid.optimizer.AdamOptimizer(learning_rate=1e-3)\n",
    "opts = optimizer.minimize(avg_cost)\n",
    "\n",
    "# 定义训练场所\n",
    "#place = fluid.CUDAPlace(0)#用GPU训练\n",
    "place = fluid.CPUPlace() #用CPU训练\n",
    "exe = fluid.Executor(place)\n",
    "# 进行参数初始化\n",
    "exe.run(fluid.default_startup_program())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "> - 接下来我们加载预训练模型，我们使用paddle官网上训练好的ResNet50模型，这个模型存储在“./ResNet50_pretrained/”，大家也可以去官网上下载，链接：http://paddle-imagenet-models-name.bj.bcebos.com/ResNet50_pretrained.tar  \n",
    "> - 我们通过if_exist函数判断网络所需的模型文件是否存在，然后再通过调用fluid.io.load_vars加载存在的模型文件。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Load model done\n"
     ]
    }
   ],
   "source": [
    "# 官方提供的原预训练模型\n",
    "src_pretrain_model_path = './ResNet50_pretrained/'\n",
    "\n",
    "\n",
    "# 通过这个函数判断模型文件是否存在\n",
    "def if_exist(var):\n",
    "    path = os.path.join(src_pretrain_model_path, var.name)\n",
    "    exist = os.path.exists(path)\n",
    "\n",
    "    return exist\n",
    "\n",
    "\n",
    "# 加载模型文件，只加载存在模型的模型文件\n",
    "fluid.io.load_vars(executor=exe, dirname=src_pretrain_model_path, predicate=if_exist, main_program=base_model_program)\n",
    "print(\"Load model done\")\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 4.开始训练\n",
    "接下来我们就定义一个双层循环来开始训练模型，并且还可以把训练过程中的cost值和acc值打印出来，以此来直观的感受我们的训练效果。\n",
    "<img src=\"images_jupyter/no_freeze_step-1.png\" width=\"800\"><br/>\n",
    "<img src=\"images_jupyter/freeze_step1.png\" width=\"850\"><br/>"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "memory_optimize is deprecated. Use CompiledProgram and Executor\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pass:0, Batch:0, Cost:1.63672, Accuracy:0.25000\n",
      "Pass:0, Batch:100, Cost:0.55392, Accuracy:0.87500\n",
      "Pass:1, Batch:0, Cost:0.54422, Accuracy:0.68750\n",
      "Pass:1, Batch:100, Cost:0.82934, Accuracy:0.56250\n",
      "Pass:2, Batch:0, Cost:0.29069, Accuracy:0.87500\n",
      "Pass:2, Batch:100, Cost:0.34632, Accuracy:0.87500\n",
      "Pass:3, Batch:0, Cost:0.35642, Accuracy:0.81250\n",
      "Pass:3, Batch:100, Cost:0.54164, Accuracy:0.81250\n",
      "Pass:4, Batch:0, Cost:0.56575, Accuracy:0.68750\n",
      "Pass:4, Batch:100, Cost:0.30929, Accuracy:0.93750\n",
      "Pass:5, Batch:0, Cost:0.43949, Accuracy:0.81250\n",
      "Pass:5, Batch:100, Cost:0.35013, Accuracy:0.93750\n",
      "Pass:6, Batch:0, Cost:0.19718, Accuracy:0.93750\n",
      "Pass:6, Batch:100, Cost:0.37463, Accuracy:0.87500\n",
      "Pass:7, Batch:0, Cost:0.19352, Accuracy:1.00000\n",
      "Pass:7, Batch:100, Cost:0.37397, Accuracy:0.93750\n",
      "Pass:8, Batch:0, Cost:0.22366, Accuracy:0.93750\n",
      "Pass:8, Batch:100, Cost:0.14132, Accuracy:0.93750\n",
      "Pass:9, Batch:0, Cost:0.36529, Accuracy:0.93750\n",
      "Pass:9, Batch:100, Cost:0.17378, Accuracy:0.87500\n"
     ]
    }
   ],
   "source": [
    "# 优化内存\n",
    "optimized = fluid.transpiler.memory_optimize(input_program=fluid.default_main_program(), print_log=False)\n",
    "\n",
    "# 定义输入数据维度\n",
    "feeder = fluid.DataFeeder(place=place, feed_list=[image, label])\n",
    "\n",
    "# 训练10次\n",
    "for pass_id in range(10):\n",
    "    # 进行训练\n",
    "    for batch_id, data in enumerate(train_reader()):\n",
    "        train_cost, train_acc = exe.run(program=fluid.default_main_program(),\n",
    "                                        feed=feeder.feed(data),\n",
    "                                        fetch_list=[avg_cost, acc])\n",
    "        # 每100个batch打印一次信息\n",
    "        if batch_id % 100 == 0:\n",
    "            print('Pass:%d, Batch:%d, Cost:%0.5f, Accuracy:%0.5f' %\n",
    "                  (pass_id, batch_id, train_cost[0], train_acc[0]))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "训练结束之后，使用fluid.io.save_param保存训练好的参数。  \n",
    "\n",
    "到这里为止，我们把从imagenet数据集上训练好的的原预训练模型，结合我们的数据集，把最后一层fc进行了训练。接下来就是使用这个已经处理过的模型正式训练了。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 保存参数模型\n",
    "save_pretrain_model_path = 'models/step-1_model/'\n",
    "\n",
    "# 删除旧的模型文件\n",
    "shutil.rmtree(save_pretrain_model_path, ignore_errors=True)\n",
    "# 创建保持模型文件目录\n",
    "os.makedirs(save_pretrain_model_path)\n",
    "# 保存参数模型\n",
    "fluid.io.save_params(executor=exe, dirname=save_pretrain_model_path)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "# step-2:\n",
    "\n",
    "经过step-1之后，如果表现不错，还想看看能不能进一步提升，就可以试试Fine-tune（即解锁比较少的卷积层继续训练），但是不要期待会有什么质的飞跃。如果由于新数据集与原数据集（例如ImageNet数据集）的差别太大导致表现很糟，那么一方面可以考虑自己从头训练模型，另一方面也可以考虑解锁比较多层的训练，亦或干脆只用预训练模型的参数作为初始值，对模型进行完整训练。  \n",
    "\n",
    "\n",
    "**但是要注意：**   \n",
    "事实上，step-2必须在已经进行过‘冻结特征提取器参数的训练’之后再尝试训练模型，这时分类器的参数已经经过一次训练。如果从随机生成的分类器参数开始直接训练，在做参数更新迭代过程中梯度将很可能过大，而导致模型的崩溃，使模型忘记学到的所有东西。\n",
    "\n",
    "### 注意：\n",
    "\n",
    "因为这是在jupyter中，所以运行下面的step-2之前，需要<font size=4>**重启一下kernel**</font>，然后执行step-2的代码，否则会报错。\n",
    "\n",
    "重启kernel方法：“网页右上角” -> 环境 -> 重启环境\n",
    "\n",
    "## 5. 导入库\n",
    "我们还是导入我们需要的库，本实验中我们专门定义了一个 reader.py文件，用来对数据集进行读取和预处理，所以我们也需要把reader.py文件import进来。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "import os\n",
    "import shutil\n",
    "import paddle as paddle\n",
    "import paddle.dataset.flowers as flowers\n",
    "import paddle.fluid as fluid\n",
    "from paddle.fluid.param_attr import ParamAttr\n",
    "import reader"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "我们利用paddle.batch()获取flowers数据集。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "done\n"
     ]
    }
   ],
   "source": [
    "# 获取flowers数据\n",
    "train_reader = paddle.batch(reader.train(), batch_size=16)\n",
    "test_reader = paddle.batch(reader.val(), batch_size=16)\n",
    "print(\"done\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 6.定义ResNet网络\n",
    "\n",
    "仍然需要定义一个残差神经网络，这个残差神经网络跟第一步时的基本一样的，只是把分类器（也就是fc层）也加进去了，这是一个完整的神经网络。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "done\n"
     ]
    }
   ],
   "source": [
    "# 定义残差神经网络（ResNet）\n",
    "def resnet50(input, class_dim):\n",
    "    def conv_bn_layer(input, num_filters, filter_size, stride=1, groups=1, act=None, name=None):\n",
    "        conv = fluid.layers.conv2d(input=input,\n",
    "                                   num_filters=num_filters,\n",
    "                                   filter_size=filter_size,\n",
    "                                   stride=stride,\n",
    "                                   padding=(filter_size - 1) // 2,\n",
    "                                   groups=groups,\n",
    "                                   act=None,\n",
    "                                   param_attr=ParamAttr(name=name + \"_weights\"),\n",
    "                                   bias_attr=False,\n",
    "                                   name=name + '.conv2d.output.1')\n",
    "        if name == \"conv1\":\n",
    "            bn_name = \"bn_\" + name\n",
    "        else:\n",
    "            bn_name = \"bn\" + name[3:]\n",
    "        return fluid.layers.batch_norm(input=conv,\n",
    "                                       act=act,\n",
    "                                       name=bn_name + '.output.1',\n",
    "                                       param_attr=ParamAttr(name=bn_name + '_scale'),\n",
    "                                       bias_attr=ParamAttr(bn_name + '_offset'),\n",
    "                                       moving_mean_name=bn_name + '_mean',\n",
    "                                       moving_variance_name=bn_name + '_variance', )\n",
    "\n",
    "    def shortcut(input, ch_out, stride, name):\n",
    "        ch_in = input.shape[1]\n",
    "        if ch_in != ch_out or stride != 1:\n",
    "            return conv_bn_layer(input, ch_out, 1, stride, name=name)\n",
    "        else:\n",
    "            return input\n",
    "\n",
    "    def bottleneck_block(input, num_filters, stride, name):\n",
    "        conv0 = conv_bn_layer(input=input,\n",
    "                              num_filters=num_filters,\n",
    "                              filter_size=1,\n",
    "                              act='relu',\n",
    "                              name=name + \"_branch2a\")\n",
    "        conv1 = conv_bn_layer(input=conv0,\n",
    "                              num_filters=num_filters,\n",
    "                              filter_size=3,\n",
    "                              stride=stride,\n",
    "                              act='relu',\n",
    "                              name=name + \"_branch2b\")\n",
    "        conv2 = conv_bn_layer(input=conv1,\n",
    "                              num_filters=num_filters * 4,\n",
    "                              filter_size=1,\n",
    "                              act=None,\n",
    "                              name=name + \"_branch2c\")\n",
    "\n",
    "        short = shortcut(input, num_filters * 4, stride, name=name + \"_branch1\")\n",
    "\n",
    "        return fluid.layers.elementwise_add(x=short, y=conv2, act='relu', name=name + \".add.output.5\")\n",
    "\n",
    "    depth = [3, 4, 6, 3]\n",
    "    num_filters = [64, 128, 256, 512]\n",
    "\n",
    "    conv = conv_bn_layer(input=input, num_filters=64, filter_size=7, stride=2, act='relu', name=\"conv1\")\n",
    "    conv = fluid.layers.pool2d(input=conv, pool_size=3, pool_stride=2, pool_padding=1, pool_type='max')\n",
    "\n",
    "    for block in range(len(depth)):\n",
    "        for i in range(depth[block]):\n",
    "            conv_name = \"res\" + str(block + 2) + chr(97 + i)\n",
    "            conv = bottleneck_block(input=conv,\n",
    "                                    num_filters=num_filters[block],\n",
    "                                    stride=2 if i == 0 and block != 0 else 1,\n",
    "                                    name=conv_name)\n",
    "\n",
    "    pool = fluid.layers.pool2d(input=conv, pool_size=7, pool_type='avg', global_pooling=True)\n",
    "    output = fluid.layers.fc(input=pool, size=class_dim, act='softmax')\n",
    "    return output\n",
    "print(\"done\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "## 7.训练前准备\n",
    "\n",
    "接下来，我们开始做训练前的准备工作：  \n",
    "\n",
    "> - 定义图片数据和标签数据的输入层;  \n",
    "> - 定义损失函数;  \n",
    "> - 求准确率；  \n",
    "> - 定义优化器；    \n",
    "> - 设定训练场所；  \n",
    "> - 定义执行器，并且完成参数初始化；"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[]"
      ]
     },
     "execution_count": 4,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 定义输入层\n",
    "image = fluid.layers.data(name='image', shape=[3, 224, 224], dtype='float32')\n",
    "label = fluid.layers.data(name='label', shape=[1], dtype='int64')\n",
    "\n",
    "# 获取分类器\n",
    "model = resnet50(image, 5)\n",
    "\n",
    "# 获取损失函数和准确率函数\n",
    "cost = fluid.layers.cross_entropy(input=model, label=label)\n",
    "avg_cost = fluid.layers.mean(cost)\n",
    "acc = fluid.layers.accuracy(input=model, label=label)\n",
    "\n",
    "# 获取训练和测试程序\n",
    "test_program = fluid.default_main_program().clone(for_test=True)\n",
    "\n",
    "# 定义优化方法\n",
    "optimizer = fluid.optimizer.AdamOptimizer(learning_rate=1e-3)\n",
    "opts = optimizer.minimize(avg_cost)\n",
    "\n",
    "\n",
    "# 定义一个使用GPU的执行器\n",
    "place = fluid.CUDAPlace(0)\n",
    "#place = fluid.CPUPlace()\n",
    "exe = fluid.Executor(place)\n",
    "# 进行参数初始化\n",
    "exe.run(fluid.default_startup_program())\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "加载经过step-1我们训练好的模型，作为新的预训练模型。并在此基础上进行重新训练："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": [
    "# 经过step-1处理后的的预训练模型\n",
    "pretrained_model_path = 'models/step-1_model/'\n",
    "\n",
    "# 加载经过处理的模型\n",
    "fluid.io.load_params(executor=exe, dirname=pretrained_model_path)\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "接下来我们就定义一个双层循环来开始训练模型，并且还可以把训练过程中的cost值和acc值打印出来，以此来直观的感受我们的训练效果。\n",
    "<img src=\"images_jupyter/no_yuxunlian.png\" width=\"750\"><br/>\n",
    "<img src=\"images_jupyter/step-1-step-2.png\" width=\"800\"><br/>\n",
    "从图上可以看出，当我们使用了预训练模型之后，效果还是很明显的，只10轮，就可以达到很高的精度了，并且训练速度很快。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Pass:0, Batch:0, Cost:0.35031, Accuracy:0.87500\n",
      "Pass:0, Batch:100, Cost:1.16975, Accuracy:0.50000\n",
      "Test:0, Cost:2.12515, Accuracy:0.53906\n",
      "Pass:1, Batch:0, Cost:1.26812, Accuracy:0.56250\n",
      "Pass:1, Batch:100, Cost:1.04485, Accuracy:0.75000\n",
      "Test:1, Cost:0.87699, Accuracy:0.71354\n",
      "Pass:2, Batch:0, Cost:0.94368, Accuracy:0.75000\n",
      "Pass:2, Batch:100, Cost:0.90288, Accuracy:0.56250\n",
      "Test:2, Cost:6.11001, Accuracy:0.56771\n",
      "Pass:3, Batch:0, Cost:0.71799, Accuracy:0.68750\n",
      "Pass:3, Batch:100, Cost:1.39456, Accuracy:0.50000\n",
      "Test:3, Cost:0.64578, Accuracy:0.78646\n",
      "Pass:4, Batch:0, Cost:0.61413, Accuracy:0.81250\n",
      "Pass:4, Batch:100, Cost:0.71937, Accuracy:0.68750\n",
      "Test:4, Cost:0.68102, Accuracy:0.72396\n",
      "Pass:5, Batch:0, Cost:1.00939, Accuracy:0.56250\n",
      "Pass:5, Batch:100, Cost:0.45485, Accuracy:0.87500\n",
      "Test:5, Cost:0.61027, Accuracy:0.80990\n",
      "Pass:6, Batch:0, Cost:0.59888, Accuracy:0.87500\n",
      "Pass:6, Batch:100, Cost:0.67762, Accuracy:0.75000\n",
      "Test:6, Cost:0.59694, Accuracy:0.78906\n",
      "Pass:7, Batch:0, Cost:0.53733, Accuracy:0.75000\n",
      "Pass:7, Batch:100, Cost:0.70325, Accuracy:0.68750\n",
      "Test:7, Cost:1.00161, Accuracy:0.70833\n",
      "Pass:8, Batch:0, Cost:0.46749, Accuracy:0.81250\n",
      "Pass:8, Batch:100, Cost:0.75960, Accuracy:0.81250\n",
      "Test:8, Cost:0.50762, Accuracy:0.82552\n",
      "Pass:9, Batch:0, Cost:0.59432, Accuracy:0.75000\n",
      "Pass:9, Batch:100, Cost:0.39056, Accuracy:0.75000\n",
      "Test:9, Cost:0.57172, Accuracy:0.82552\n"
     ]
    }
   ],
   "source": [
    "# 定义输入数据维度\n",
    "feeder = fluid.DataFeeder(place=place, feed_list=[image, label])\n",
    "\n",
    "# 训练10次\n",
    "for pass_id in range(10):\n",
    "    # 进行训练\n",
    "    for batch_id, data in enumerate(train_reader()):\n",
    "        train_cost, train_acc = exe.run(program=fluid.default_main_program(),\n",
    "                                        feed=feeder.feed(data),\n",
    "                                        fetch_list=[avg_cost, acc])\n",
    "        # 每100个batch打印一次信息\n",
    "        if batch_id % 100 == 0:\n",
    "            print('Pass:%d, Batch:%d, Cost:%0.5f, Accuracy:%0.5f' %\n",
    "                  (pass_id, batch_id, train_cost[0], train_acc[0]))\n",
    "\n",
    "    # 进行测试\n",
    "    test_accs = []\n",
    "    test_costs = []\n",
    "    for batch_id, data in enumerate(test_reader()):\n",
    "        test_cost, test_acc = exe.run(program=test_program,\n",
    "                                      feed=feeder.feed(data),\n",
    "                                      fetch_list=[avg_cost, acc])\n",
    "        test_accs.append(test_acc[0])\n",
    "        test_costs.append(test_cost[0])\n",
    "    # 求测试结果的平均值\n",
    "    test_cost = (sum(test_costs) / len(test_costs))\n",
    "    test_acc = (sum(test_accs) / len(test_accs))\n",
    "    print('Test:%d, Cost:%0.5f, Accuracy:%0.5f' % (pass_id, test_cost, test_acc))\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {
    "collapsed": false
   },
   "source": [
    "训练结束之后，可以保存预测模型用于之后的预测使用。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {
    "collapsed": false
   },
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['save_infer_model/scale_0']"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# 保存预测模型\n",
    "save_path = 'models/step_2_model/'\n",
    "\n",
    "# 删除旧的模型文件\n",
    "shutil.rmtree(save_path, ignore_errors=True)\n",
    "# 创建保持模型文件目录\n",
    "os.makedirs(save_path)\n",
    "# 保存预测模型\n",
    "fluid.io.save_inference_model(save_path, feeded_var_names=[image.name], target_vars=[model], executor=exe)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {
    "collapsed": false
   },
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "PaddlePaddle 1.4.1 (Python 3.5)",
   "language": "python",
   "name": "py35-paddle1.2.0"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.5.5"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 1
}
